Obtaining input for controlling execution of a game program

ABSTRACT

Methods of obtaining input for controlling execution of a game program are disclosed.

CLAIM OF PRIORITY

This application also claims benefit of U.S. Provisional PatentApplication No. 60/718,145, entitled “AUDIO, VIDEO, SIMULATION, AND USERINTERFACE PARADIGMS”, filed Sep. 15, 2005, which is hereby incorporatedby reference.

This application is a continuation in part (CIP) of U.S. patentapplication Ser. No. 10/207,677, entitled, “MAN-MACHINE INTERFACE USINGA DEFORMABLE DEVICE”, filed on Jul. 27, 2002; U.S. patent applicationSer. No. 10/650,409, entitled, “AUDIO INPUT SYSTEM”, filed on Aug. 27,2003; U.S. patent application Ser. No. 10/663,236, entitled “METHOD ANDAPPARATUS FOR ADJUSTING A VIEW OF A SCENE BEING DISPLAYED ACCORDING TOTRACKED HEAD MOTION”, filed on Sep. 15, 2003; U.S. patent applicationSer. No. 10/759,782, entitled “METHOD AND APPARATUS FOR LIGHT INPUTDEVICE”, filed on Jan. 16, 2004; U.S. patent application Ser. No.10/820,469, entitled “METHOD AND APPARATUS TO DETECT AND REMOVE AUDIODISTURBANCES”, filed on Apr. 7, 2004; and U.S. patent application Ser.No. 11/301,673, entitled “METHOD FOR USING RELATIVE HEAD AND HANDPOSITIONS TO ENABLE A POINTING INTERFACE VIA CAMERA TRACKING”, filed onDec. 12, 2005, all of which are hereby incorporated by reference.

This application is also a continuation in part (CIP) of U.S. patentapplication Ser. No. 11/381,729, to Xiao Dong Mao, entitled ULTRA SMALLMICROPHONE ARRAY, (Attorney Docket SCEA05062US00), filed on May 4, 2006,application Ser. No. 11/381,728, to Xiao Dong Mao, entitled ECHO ANDNOISE CANCELLATION, (Attorney Docket SCEA05064US00), filed on May 4,2006, U.S. patent application Ser. No. 11/381,725, to Xiao Dong Mao,entitled “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION”, (AttorneyDocket SCEA05072US00), filed on May 4, 2006, U.S. patent applicationSer. No. 11/381,727, to Xiao Dong Mao, entitled “NOISE REMOVAL FORELECTRONIC DEVICE WITH FAR FIELD MICROPHONE ON CONSOLE”, (AttorneyDocket SCEA05073US00), filed on May 4, 2006, U.S. patent applicationSer. No. 11/381,724, to Xiao Dong Mao, entitled “METHODS AND APPARATUSFOR TARGETED SOUND DETECTION AND CHARACTERIZATION”, (Attorney DocketSCEA05079US00), filed on May 4, 2006, U.S. patent application Ser. No.11/381,721, to Xiao Dong Mao, entitled “SELECTIVE SOUND SOURCE LISTENINGIN CONJUNCTION WITH COMPUTER INTERACTIVE PROCESSING”, (Attorney DocketSCEA04005JUMBOUS), filed on May 4, 2006; all of which are herebyincorporated by reference.

This application is also a continuation in part (CIP) of: co-pendingapplication Ser. No. ______, to Xiao Dong Mao, entitled “METHODS ANDAPPARATUSES FOR ADJUSTING A LISTENING AREA FOR CAPTURING SOUNDS”,(Attorney Docket SCEA-00300) filed on May 4, 2006; co-pendingapplication Ser. No. ______, to Xiao Dong Mao, entitled “METHODS ANDAPPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON VISUAL IMAGE”,(Attorney Docket SCEA-00400), filed on May 4, 2006; co-pendingapplication Ser. No. ______, to Xiao Dong Mao, entitled “METHODS ANDAPPARATUSES FOR CAPTURING AN AUDIO SIGNAL BASED ON A LOCATION OF THESIGNAL”, (Attorney Docket SCEA-00500), filed on May 4, 2006; co-pendingapplication Ser. No. ______, to Richard Marks et al., entitled“SELECTIVE SOUND SOURCE LISTENING IN CONJUNCTION WITH COMPUTERINTERACTIVE PROCESSING”, (Attorney Docket SCEA04005US01-SONYP045), filedon May 4, 2006; and co-pending application Ser. No. ______, to RichardMarks et al., entitled “Computer Image and Audio Processing of Intensityand Input Devices for Interfacing With A Computer Program”, (AttorneyDocket SONYP052), filed on May 4, 2006, all of the entire disclosures ofwhich are incorporated herein by reference.

This application is also a continuation in part (CIP) of U.S. patentapplication Ser. No. 11/382,031, entitled “MULTI-INPUT GAME CONTROLMIXER”, (Attorney Docket SCEA06MXR1), filed on May 6, 2006; U.S. patentapplication Ser. No. 11/382,032, entitled “SYSTEM FOR TRACKING USERMANIPULATIONS WITHIN AN ENVIRONMENT”, (Attorney Docket SCEA06MXR2),filed on May 6, 2006; U.S. patent application Ser. No. 11/382,033,entitled “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUTCONTROL”, (Attorney Docket SCEA061NRT1), filed on May 6, 2006; U.S.patent application Ser. No. 11/382,035, entitled “INERTIALLY TRACKABLEHAND-HELD CONTROLLER”, (Attorney Docket SCEA061NRT2), filed on May 6,2006; U.S. patent application Ser. No. 11/382,036, entitled “METHOD ANDSYSTEM FOR APPLYING GEARING EFFECTS TO VISUAL TRACKING”, (AttorneyDocket SONYP058A), filed on May 6, 2006; U.S. patent application Ser.No. 11/382,041, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTSTO INERTIAL TRACKING”, (Attorney Docket SONYP058B), filed on May 7,2006; U.S. patent application Ser. No. 11/382,038, entitled “METHOD ANDSYSTEM FOR APPLYING GEARING EFFECTS TO ACOUSTICAL TRACKING”, (AttorneyDocket SONYP058C), filed on May 6, 2006; U.S. patent application Ser.No. 11/382,040, entitled “METHOD AND SYSTEM FOR APPLYING GEARING EFFECTSTO MULTI-CHANNEL MIXED INPUT”, (Attorney Docket SONYP058D), filed on May7, 2006; U.S. patent application Ser. No. 11/382,034, entitled “SCHEMEFOR DETECTING AND TRACKING USER MANIPULATION OF A GAME CONTROLLER BODY”,(Attorney Docket 86321 SCEA05082US00), filed on May 6, 2006; U.S. patentapplication Ser. No. 11/382,037, entitled “SCHEME FOR TRANSLATINGMOVEMENTS OF A HAND-HELD CONTROLLER INTO INPUTS FOR A SYSTEM”, (AttorneyDocket 86324), filed on May 6, 2006; U.S. patent application Ser. No.11/382,043, entitled “DETECTABLE AND TRACKABLE HAND-HELD CONTROLLER”,(Attorney Docket 86325), filed on May 7, 2006; U.S. patent applicationSer. No. 11/382,039, entitled “METHOD FOR MAPPING MOVEMENTS OF AHAND-HELD CONTROLLER TO GAME COMMANDS”, (Attorney Docket 86326), filedon May 7, 2006; U.S. Design patent application Ser. No. ______, entitled“CONTROLLER WITH INFRARED PORT”, (Attorney Docket SCEA06007US00), filedon May 6, 2006; U.S. Design patent application Ser. No. ______, entitled“CONTROLLER WITH TRACKING SENSORS”, (Attorney Docket SCEA06008US00),filed on May 6, 2006; U.S. patent application Ser. No. ______, entitled“DYNAMIC TARGET INTERFACE”, (Attorney Docket SCEA06009US00), filed onMay 6, 2006; and U.S. Design patent application Ser. No. ______,entitled “TRACKED CONTROLLER DEVICE”, (Attorney Docket SCEA06010US00),filed on May 6, 2006; all of which are hereby incorporated herein byreference in their entireties.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is also related to co-pending U.S. patent applicationSer. No. ______, to Gary Zalewski and Riley R. Russell, entitled“Profile Detection”, (Attorney Docket SCEA05059US00), filed on the sameday as this application, the entire disclosure of which is incorporatedherein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, to Gary Zalewski and Riley R. Russell, entitled “UsingAudio/Visual Environment To Select Ads On Game Platform”, (AttorneyDocket SCEAJP 3.0-003 CIP V), filed on the same day as this application,the entire disclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. 11/400,997, filed on Apr. 10, 2006, to Larsen and Chen,entitled “System And Method For Obtaining User Information From Voices”,(Attorney Docket SCEA05040US00), the entire disclosure of which isincorporated herein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, to Gary Zalewski et al., entitled “Method and apparatusfor use in determining lack of user activity in relation to a system”,(Attorney Docket 86327), filed on the same day as this application, theentire disclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, Gary Zalewski et al., entitled “Method and apparatusfor use in determining an activity level of a user in relation to asystem”, (Attorney Docket 86328), filed on the same day as thisapplication, the entire disclosure of which is incorporated herein byreference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, to Gary Zalewski et al., entitled “Hand-held controllerhaving detectable elements for tracking purposes”, (Attorney Docket86329), filed on the same day as this application, the entire disclosureof which is incorporated herein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, entitled “TRACKING DEVICE FOR USE IN OBTAININGINFORMATION FOR CONTROLLING GAME PROGRAM EXECUTION”, (Attorney DocketSCEA061NRT3), filed on the same day as this application, the entiredisclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. patent applicationSer. No. ______, entitled “TRACKING DEVICE WITH SOUND EMITTER FOR USE INOBTAINING INFORMATION FOR CONTROLLING GAME PROGRAM EXECUTION”, (AttorneyDocket SCEA06ACRA2), filed on the same day as this application, theentire disclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. 29/246,744, entitled “VIDEO GAME CONTROLLER FRONTFACE”, (Attorney Docket SCEACTR-D3), filed on the same day as thisapplication, the entire disclosure of which is incorporated herein byreference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. 29/246,743, entitled “VIDEO GAME CONTROLLER”,(Attorney Docket SCEACTRL-D2), filed on the same day as thisapplication, the entire disclosure of which is incorporated herein byreference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “VIDEO GAME CONTROLLER”, (AttorneyDocket SONYP059A), filed on the same day as this application, the entiredisclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “VIDEO GAME CONTROLLER”, (AttorneyDocket SONYP059B), filed on the same day as this application, the entiredisclosure of which is incorporated herein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “ERGONOMIC GAME CONTROLLER DEVICEWITH LEDS AND OPTICAL PORTS”, (Attorney Docket PA3760US), filed on thesame day as this application, the entire disclosure of which isincorporated herein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “GAME CONTROLLER DEVICE WITH LEDSAND OPTICAL PORTS”, (Attorney Docket PA3761US), filed on the same day asthis application, the entire disclosure of which is incorporated hereinby reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “DESIGN FOR OPTICAL GAMECONTROLLER INTERFACE”, (Attorney Docket PA3762US), filed on the same dayas this application, the entire disclosure of which is incorporatedherein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “DUAL GRIP GAME CONTROL DEVICEWITH LEDS AND OPTICAL PORTS”, (Attorney Docket PA3763US), filed on thesame day as this application, the entire disclosure of which isincorporated herein by reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “GAME INTERFACE DEVICE WITH LEDSAND OPTICAL PORTS”, (Attorney Docket PA3764US), filed on the same day asthis application, the entire disclosure of which is incorporated hereinby reference.

This application is also related to co-pending U.S. Design patentapplication Ser. No. ______, entitled “ERGONOMIC GAME INTERFACE DEVICEWITH LEDS AND OPTICAL PORTS”, (Attorney Docket PA3765US), filed on thesame day as this application, the entire disclosure of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to human-computer interfacingand specifically to processing multi-channel input for tracking the usermanipulation of one or more controllers.

BACKGROUND OF THE INVENTION

Computer entertainment systems typically include a hand-held controller,game controller, or other controller. A user or player uses thecontroller to send commands or other instructions to the entertainmentsystem to control a video game or other simulation being played. Forexample, the controller may be provided with a manipulator which isoperated by the user, such as a joy stick. The manipulated variable ofthe joy stick is converted from an analog value into a digital value,which is sent to the game machine main frame. The controller may also beprovided with buttons that can be operated by the user.

It is with respect to these and other background information factorsthat the present invention has evolved.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a pictorial diagram illustrating a video game system thatoperates in accordance with an embodiment of the present invention;

FIG. 2 is a perspective view of a controller made in accordance with anembodiment of the present invention;

FIG. 3 is a three-dimensional schematic diagram illustrating anaccelerometer that may be used in a controller according to anembodiment of the present invention;

FIG. 4 is a block diagram of a system for mixing various control inputsaccording to an embodiment of the present invention;

FIG. 5A is a block diagram of a portion of the video game system of FIG.1.

FIG. 5B is a flow diagram of a method for tracking a controller of avideo game system according to an embodiment of the present invention;

FIG. 5C is a flow diagram illustrating a method for utilizing positionand/or orientation information during game play on a video game systemaccording to an embodiment of the present invention.

FIG. 6 is a block diagram illustrating a video game system according toan embodiment of the present invention; and

FIG. 7 is a block diagram of a cell processor implementation of thevideo game system according to an embodiment of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

Various embodiments of the methods, apparatus, schemes and systemsdescribed herein provide for the detection, capture and tracking of themovements, motions and/or manipulations of the entire controller bodyitself by the user. The detected movements, motions and/or manipulationsof the entire controller body by the user may be used as additionalcommands to control various aspects of the game or other simulationbeing played.

Detecting and tracking a user's manipulations of a game controller bodymay be implemented in different ways. For example, an inertial sensor,such as an accelerometer or gyroscope, an image capture unit, such as adigital camera, can be used with the computer entertainment system todetect motions of the hand-held controller body and transfer them intoactions in a game. Examples of tracking a controller with an inertialsensor are described, e.g., in U.S. patent application Ser. No. ______,entitled “SYSTEM, METHOD, AND APPARATUS FOR THREE-DIMENSIONAL INPUTCONTROL” (Attorney Docket Number SCEA061NRT1), which is incorporatedherein by reference. Examples of tracking a controller using imagecapture are described, e.g., in U.S. patent application Ser. No. ______,entitled “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAMECONTROLLER BODY” (Attorney Docket Number SCEA05082US00), which isincorporated herein by reference. In addition, the controller and/or theuser may also be tracked acoustically using a microphone array andappropriate signal processing. Examples of such acoustic tracking aredescribed in U.S. patent application Ser. No. 11/381,721, which isincorporated herein by reference.

Acoustic sensing, inertial sensing and image capture can be usedindividually or in any combination to detect many different types ofmotions of the controller, such as for example up and down movements,twisting movements, side to side movements, jerking movements, wand-likemotions, plunging motions, etc. Such motions may correspond to variouscommands such that the motions are transferred into actions in a game.Detecting and tracking the user's manipulations of a game controllerbody can be used to implement many different types of games,simulations, etc., that allow the user to, for example, engage in asword or lightsaber fight, use a wand to trace the shape of items,engage in many different types of sporting events, engage in on-screenfights or other encounters, etc. A game program may configured to trackthe motion of the controller and recognize certain pre-recorded gesturesfrom the tracked motion. Recognition of one or more of these gesturesmay trigger a change in the game state.

In embodiments of the present invention controller path informationobtained from these different sources may be mixed prior to analysis forgesture recognition. The tracking data from different sources (e.g.,acoustic, inertial and image capture) may be mixed in a way thatimproves the likelihood of recognition of a gesture.

Referring to FIG. 1, there is illustrated a system 100 that operates inaccordance with an embodiment of the present invention. As illustrated,a computer entertainment console 102 may be coupled to a television orother video display 104 to display the images of the video game or othersimulation thereon. The game or other simulation may be stored on a DVD,CD, flash memory, USB memory, or other memory media 106 that is insertedinto the console 102. A user or player 108 manipulates a a gamecontroller 110 to control the video game or other simulation. As seen inFIG. 2, the game controller 110 includes an inertial sensor 112 thatproduces signals in response to the position, motion, orientation orchange in orientation of the game controller 110. In addition to theinertial sensor, the game controller 110 may include conventionalcontrol input devices, e.g., joysticks 111, buttons 113, R1, L1, and thelike.

During operation, the user 108 physically moves the controller 110. Forexample, the controller 110 may be moved in any direction by the user108, such as up, down, to one side, to the other side, twisted, rolled,shaken, jerked, plunged, etc. These movements of the controller 110itself may be detected and captured by the camera 112 by way of trackingthrough analysis of signals from the inertial sensor 112 in a mannerdescribed below.

Referring again to FIG. 1, the system 100 may optionally include acamera or other video image capturing device 114, which may bepositioned so that the controller 110 is within the camera's field ofview 116. Analysis of images from the image capturing device 114 may beused in conjunction with analysis of data from the inertial sensor 112.As shown in FIG. 2, the controller 110 may optionally be equipped withlight sources such as light emitting diodes (LEDs) 202, 204, 206, 208 tofacilitate tracking by video analysis. These may be mounted to a body ofthe controller 110. As used herein, the term “body” meant to describethe part of the game controller 110 that one would hold by hand (or wearif it were a wearable game controller).

Analysis of such video images for the purpose of tracking the controller110 are described, e.g., in U.S. patent application Ser. No. ______,entitled “SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OF A GAMECONTROLLER BODY” (Attorney Docket Number SCEA05082US00), to inventors______, which is incorporated herein by reference. The console 110 mayinclude an acoustic transducer, such as a microphone array 118. Thecontroller 110 may also include an acoustic signal generator 210 (e.g.,a speaker) to provide a source of sound to facilitate acoustic trackingof the controller 110 with the microphone array 118 and appropriateacoustic signal processing, e.g., as described in U.S. patentapplication Ser. No. 11/381,724, which is incorporated herein byreference.

In general, signals from the inertial sensor 112 are used to generateposition and orientation data for the controller 110. Such data may beused to calculate many physical aspects of the movement of thecontroller 110, such as for example its acceleration and velocity alongany axis, its tilt, pitch, yaw, roll, as well as any telemetry points ofthe controller 110. As used herein, telemetry generally refers to remotemeasurement and reporting of information of interest to a system or tothe system's designer or operator.

The ability to detect and track the movements of the controller 110makes it possible to determine whether any predefined movements of thecontroller 110 are performed. That is, certain movement patterns orgestures of the controller 110 may be predefined and used as inputcommands for the game or other simulation. For example, a plungingdownward gesture of the controller 110 may be defined as one command, atwisting gesture of the controller 110 may be defined as anothercommand, a shaking gesture of the controller 110 may be defined asanother command, and so on. In this way the manner in which the user 108physically moves the controller 110 is used as another input forcontrolling the game, which provides a more stimulating and entertainingexperience for the user.

By way of example and without limitation, the inertial sensor 112 may bean accelerometer. FIG. 3 depicts an example of an accelerometer 300 inthe form of a simple mass 302 elastically coupled at four points to aframe 304, e.g., by springs 306, 308, 310, 312. Pitch and roll axes(indicated by X and Y, respectively) lie in a plane that intersects theframe. A yaw axis Z is oriented perpendicular to the plane containingthe pitch axis X and the roll axis Y. The frame 304 may be mounted tothe controller 110 in any suitable fashion. As the frame 304 (and thegame controller 110) accelerates and/or rotates the mass 302 maydisplace relative to the frame 304 and the springs 306, 208, 310, 312may elongate or compress in a way that depends on the amount anddirection of translational and/or rotational acceleration and/or theangle of pitch and/or roll and/or yaw. The displacement and of the mass302 and/or compression or elongation of the springs 306, 308, 310, 312may be sensed, e.g., with appropriate sensors 314, 316, 318, 320 andconverted to signals that depend in known or determinable way on theamount acceleration of pitch and/or roll.

There are a number of different ways to track the position of the massand/or or the forces exerted on it, including resistive strain gaugematerial, photonic sensors, magnetic sensors, hall-effect devices,piezoelectric devices, capacitive sensors, and the like. Embodiments ofthe invention may include any number and type or combination of types ofsensors. By way of example, and without limitation, the sensors 314,316, 318, 320 may be gap closing electrodes placed above the mass 302. Acapacitance between the mass and each electrode changes as the positionof the mass changes relative to each electrode. Each electrode may beconnected to a circuit that produce a signal related to the capacitance(and therefore to the proximity) of the mass 302 relative to theelectrode. In addition, the springs 306, 308, 310, 312 may includeresistive strain gauge sensors that produce signals that are related tothe compression or elongation of the springs.

In some embodiments, the frame 304 may be gimbal mounted to thecontroller 110 so that the accelerometer 300 maintains a fixedorientation with respect to the pitch and/or roll and/or yaw axes. Insuch a manner, the controller axes X, Y, Z may be directly mapped tocorresponding axes in real space without having to take into account atilting of the controller axes with respect to the real space coordinateaxes.

As discussed above, data from inertial, image capture and acousticsources may be analyzed to generate a path that tracks the positionand/or orientation of the controller 110. As shown in the block diagramof FIG. 4 a system 400 according to an embodiment of the invention mayinclude an inertial analyzer 402, an image analyzer 404 and an acousticanalyzer 406. Each of these analyzers receives signals from a sensedenvironment 401. The analyzers 402, 406, 408 may be implemented inhardware, in software (or firmware) or some combination of two or moreof these. Each of the analyzers produces tracking information relatedthe position and/or orientation of an object of interest. By way ofexample, the object of interest may be the controller 110 referred toabove. The image analyzer 404 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. ______ (Attorney Docket Number 86321SCEA05082US00).The inertial analyzer 402 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. ______, entitled “SYSTEM, METHOD, AND APPARATUS FORTHREE-DIMENSIONAL INPUT CONTROL” (Attorney Docket Number SCEA061NRT1).The acoustic analyzer 406 may operate in connection with and to formfields below and with respect to methods described in U.S. patentapplication Ser. No. 11/381,724.

The analyzers 402, 404 and 406 may be regarded as being associated withdifferent channels of inputs of position and/or orientation information.The Mixer 408 may accept multiple input channels and such channels maycontain sample data characterizing the sensed environment 401, typicallyfrom the perspective of the channel. The position and/or orientationinformation generated by the inertial analyzer 402, image analyzer 404and acoustic analyzer 406 can be coupled into the input of a mixer 408.The Mixer 408 and analyzers 402, 404, 406 may be queried by a gamesoftware 410 and may be configured to interrupt game software inresponse to events. Events may include gesture recognition events,gearing changes, configuration changes, setting noise levels, settingsampling rate, changing mapping chains, etc, examples of which arediscussed below. The mixer 408 may operate in connection with and toform fields below and with respect to methods described herein.

As discussed above, signals from different input channels, e.g.,inertial sensors, video images and/or acoustic sensors may be analyzedby the inertial analyzer 402, image analyzer 404 and acoustic analyzer406, respectively, to determine the motion and/or orientation of thecontroller 110 during play of a video game according to an inventivemethod. Such a method may be implemented as a series of processorexecutable program code instructions stored in a processor readablemedium and executed on a digital processor. For example, as depicted inFIG. 5A, the video game system 100 may include on the console 102 havingthe inertial analyzer 402, image analyzer 404 and acoustic analyzer 406implemented either in hardware or software. By way of example, theanalyzers 402, 404, 406 may be implemented as software instructionsrunning on a suitable processor unit 502. By way of example, theprocessor unit 502 may be a digital processor, e.g., a microprocessor ofa type commonly used in video game consoles. A portion of theinstructions may be stored in a memory 506. Alternatively, the inertialanalyzer 402, image analyzer 404 and acoustic analyzer 406 may beimplemented in hardware, e.g., as an application specific integratedcircuit (ASIC). Such analyzer hardware may be located on the controller110 or on the console 102 or may be remotely located elsewhere. Inhardware implementations, the analyzers 402, 404, 406 may beprogrammable in response to external signals e.g., from the processor502 or some other remotely located source, e.g., connected by USB cable,wireless connection, or over a network.

The inertial analyzer 402 may include or implement instructions thatanalyze the signals generated by the inertial sensor 112 and utilizeinformation regarding position and/or orientation of the controller 110.Similarly, the image analyzer 404 may implement instructions thatanalyze images captured by the image capture unit 114. In addition, theacoustic analyzer may implement instructions that analyze imagescaptured by the microphone array 118. As shown in the flow diagram 510of FIG. 5B these signals and/or images may be received by the analyzers402, 404, 406 as indicated at block 512. The signals and/or images maybe analyzed by the analyzers 402, 404, 406 to determine inertialtracking information 403, image tracking information 405 and acoustictracking information 407 regarding the position and/or orientation ofthe controller 110 as indicated at block 514. The tracking information403, 405, 407 may be related to one or more degrees of freedom. It ispreferred that six degrees of freedom are tracked to characterize themanipulation of the controller 110 or other tracked object. Such degreesof freedom may relate to the controller tilt, yaw, roll and position,velocity or acceleration along the x, y and z-axis.

As indicated at block 516, the mixer 408 mixes the inertial information403, image information 405 and acoustic information 407 to generaterefined position and/or orientation information 409. By way of example,the mixer 408 may apply different weights the inertial, image andacoustic tracking information 403, 405, 407 based on game orenvironmental conditions and the take a weighted average. In addition,the mixer 408 may include its own mixer analyzer 412 that analyzes thecombined position/orientation information and generates its ownresulting “mixer” information that involves combinations of theinformation generated by the other analyzers.

In an embodiment of the present invention the mixer 408 may assign adistribution value to the tracking information 403, 405, 407 from theanalyzers 402, 404, 406. As noted above, certain sets of input controldata may be averaged. In the present embodiment, however, the inputcontrol data is assigned a value prior to its being averaged whereby theinput control data from some analyzers is of more analytical importancethan from others.

The mixer 408 may take on a number of functionalities in the context ofthe present system including observation, correction, stabilization,derivation, combination, routing, mixing, reporting, buffering,interrupting other processes and analysis. Such may be performed withrespect to the tracking information 403, 405, 407 received from one ormore of the analyzers 402, 404, 406. While each of the analyzers 402,404, 406 may receive and/or derive certain tracking information, themixer 408 may be implemented to optimize the use of the receivedtracking information 403, 405, 407 and generate refined trackinginformation 409.

The analyzers 402, 404, 406 and mixer 408 are preferably configured toprovide tracking information similar output formats. Trackinginformation parameters from any analyzer element 402, 404, 406 can bemapped to a single parameter in an analyzer. Alternatively, the mixer408 may form tracking information for any of the analyzers 402, 404, 406by processing one or more tracking information parameters from one ormore of analyzers 402, 404, 406. The mixer may combine two or moreelements of tracking information of the same parameter type taken fromthe analyzers 402, 404, 406 and/or perform functions across multipleparameters of tracking information generated by the analyzers to createa synthetic set of output having the benefit of being generated frommultiple channels of input.

The refined tracking information 409 may be utilized during play of avideo game with the system 100 as indicated at block 518. In certainembodiments, the position and/or orientation information may be used inrelation to gestures made by the user 108 during game play. In someembodiments the mixer 408 may operate in conjunction with the gesturerecognizer 505 to associate at least one action in a game environmentwith one or more user actions from the user (e.g., manipulation of thecontroller in space).

As indicated in the flow diagram 520 of FIG. 5C, a path of thecontroller 110 may be tracked using the position and/or orientationinformation as indicated at block 522. By way of example, and withoutlimitation, the path may include a set of points representing a positionof the center of mass of the controller with respect to some system ofcoordinates. Each position point may be represented by one or morecoordinates, e.g., X, Y and Z coordinates in a Cartesian coordinatesystem. A time may be associated with each point on the path so thatboth the shape of the path and the progress of the controller along thepath may be monitored. In addition, each point in the set may haveassociated with it data representing an orientation of the controller,e.g., one or more angles of rotation of the controller about its centerof mass. Furthermore, each point on the path may have associated with itvalues of velocity and acceleration of the center of mass of thecontroller and rates of angular rotation and angular acceleration of thecontroller about its center of mass.

As indicated at block 524, the tracked path may be compared to one ormore stored paths corresponding to known and/or pre-recorded gestures508 that are relevant to the context of the video game being played.Recognizer 505 may be configured to recognize a user or process audioauthenticated gestures, etc. For example, a user may be identified bythe recognizer 505 through a gesture and that a gesture may be specificto a user. Such a specific gestures may recorded and included among thepre-recorded gestures 508 stored in memory 506. The recordation processmay optionally store audio generated during recordation of a gesture.The sensed environment is sampled into a multi-channel analyzer andprocessed. The processor may reference gesture models to determine andauthenticate and/or identify a user or objects based on voice oracoustic patterns and to a high degree of accuracy and performance.

As indicated in FIG. 5A, data 508 representing the gestures may bestored in the memory 506. Examples of gestures include, but are notlimited to throwing an object such as a ball, swinging an object such asa bat or golf club, pumping hand pump, opening or closing a door orwindow, turning steering wheel or other vehicle control, martial artsmoves such as punches, sanding movements, wax on wax off, paint thehouse, shakes, rattles, rolls, football pitches, turning knob movements,3D MOUSE movements, scrolling movements, movements with known profiles,any recordable movement, movements along any vector back and forth i.e.pump the tire but at some arbitrary orientation in space, movementsalong a path, movements having precise stop and start times, any timebased user manipulation that can be recorded, tracked and repeatedwithin the noise floor, splines, and the like. Each of these gesturesmay be pre-recorded from path data and stored as a time-based model.Comparison of the path and stored gestures may start with an assumptionof a steady state if the path deviates from a steady state the path canbe compared to the stored gestures by a process of elimination. If atblock 526 there is no match, the analyzer 504 may continue tracking thepath of the controller 110 at block 522. If there is a sufficient matchbetween the path (or a portion thereof) and a stored gesture the stateof the game may be changed as indicated at 528. Changes of state of thegame may include, but are not limited to interrupts, sending controlsignals, changing variables, etc.

Here is one example of this can occur. Upon determining that thecontroller 110 has left a steady state the path an analyzer 402, 404,406, or 412 tracks movement of the controller 110. As long as the pathof the controller 110 complies with a path defined in the stored gesturemodels 508, those gestures are possible “hits”. If the path of thecontroller 110 deviates (within the noise tolerance setting) from anygesture model 508, that gesture model is removed from the hit list. Eachgesture reference model includes a time-base in which the gesture isrecorded. The analyzer 402, 404, 406, or 412 compares the controllerpath data to the stored gestures 508 at the appropriate time index.Occurrence of a steady state condition resets the clock. When deviatingfrom steady state (i.e. when movements are tracked outside of the noisethreshold) the hit list is populated with all potential gesture models.The clock is started and movements of the controller are comparedagainst the hit list. Again, the comparison is a walk through time. Ifany gesture in the hit list reaches the end of the gesture then it is ahit.

In certain embodiments, the mixer 408 and/or individual analyzers 402,404, 406, 412 may inform a game program when certain events occur.Examples of such events include the following:

INTERRUPT ZERO-ACCELERATION POINT REACHED (X AND/OR Y AN/OR Z AXIS) Incertain game situations the analyzer 504 may notify or interrupt routinewithin the game program when acceleration of the controller changes atthe inflection points. For example, the user 108 may use the controller110 to control a game avatar representing a quarterback in a footballsimulation game. The analyzer 504 may track the controller (representingthe football) via a path generated from signals from the inertial sensor112. A particular change in acceleration of the controller 110 maysignal release of the football. At this point, the analyzer may triggeranother routine within the program (e.g., a physics simulation package)to simulate the trajectory of the football based on the position, and/orvelocity and/or orientation of the controller at the point of release.

Interrupt New Gesture Recognized

In addition, the analyzer 502 may be configured by one or more inputs.Examples of such inputs include, but are not limited to:

SET NOISE LEVEL (X, Y or Z AXIS) The noise level may be a referencetolerance used when analyzing jitter of the user's hands in the game.

SET SAMPLING RATE. As used herein, the sampling rate may refer to howoften the analyzer 502 samples the signals from the inertial sensor. Thesampling rate may be set to oversample or average the signal.

SET GEARING. As used herein gearing generally refers to the ratio ofcontroller movements to movements occurring within the game. Examples ofsuch “gearing” in the context of control of a video game may be found inU.S. patent application Ser. No. ______, filed ______, (Attorney DocketNo.: ______), which is incorporated herein by reference.

SET MAPPING CHAIN. As used herein, a mapping chain refers to a map ofgesture models. The gesture model maps can be made for a specific inputChannel (e.g., for path data generated from inertial sensor signalsonly) or for a hybrid Channel formed in a mixer unit. Three inputChannels may be served by two or more different Analyzers that aresimilar to the inertial analyzer 504. Specifically, these may include:the inertial analyzer 504 as described herein, a video analyzer asdescribed e.g., in U.S. patent application Ser. No. ______, to inventors______, entitled SCHEME FOR DETECTING AND TRACKING USER MANIPULATION OFA GAME CONTROLLER BODY (Attorney Docket SCEA05082US00), which isincorporated herein by reference, and an Acoustic Analyzer, e.g., asdescribed in U.S. patent application Ser. No. 11/381,721, which isincorporated herein by reference. The Analyzers can be configured with amapping chain. Mapping chains can be swapped out by the game duringgameplay as can settings to the Analyzer and to the Mixer.

Referring to again to FIG. 5A, block 502, those of skill in the art willrecognize that there are numerous ways to generate signals from theinertial sensor 112. A few examples, among others have been describedabove with respect to FIGS. 3A-3E. Referring to block 504, there arenumerous ways to analyze the sensor signals generated in block 502 toobtain information relating to the position and/or orientation of thecontroller 110. By way of example and without limitation the trackinginformation may include, but is not limited to information regarding thefollowing parameters individually or in any combination:

CONTROLLER ORIENTATION. Orientation of the controller 110 may beexpressed in terms of pitch, roll or yaw angle with respect to somereference orientation, e.g., in radians). Rates of change of controllerorientation (e.g., angular velocities or angular accelerations) may alsobe included in the position and/or orientation information. Where theinertial sensor 112 includes a gyroscopic sensor controller orientationinformation may be obtained directly in the form of one or more outputvalues that are proportional to angles of pitch, roll or yaw.

CONTROLLER POSITION (e.g., Cartesian coordinates X,Y,Z of the controller110 in some frame of reference)

CONTROLLER X-AXIS VELOCITY

CONTROLLER Y-AXIS VELOCITY

CONTROLLER Z-AXIS VELOCITY

CONTROLLER X-AXIS ACCELERATION

CONTROLLER Y-AXIS ACCELERATION

CONTROLLER Z-AXIS ACCELERATION

It is noted that with respect to position, velocity and acceleration theposition and/or orientation information may be expressed in terms ofcoordinate systems other than Cartesian. For example, cylindrical orspherical coordinates may be used for position, velocity andacceleration. Acceleration information with respect to the X, Y and Zaxes may be obtained directly from an accelerometer type sensor, e.g.,as described above with respect to FIGS. 3A-3E. The X, Y and Zaccelerations may be integrated with respect to time from some initialinstant to determine changes in X, Y and Z velocities. These velocitiesmay be computed by adding the velocity changes to known values of theX-, Y-, and Z-velocities at the initial instant in time. The X, Y and Zvelocities may be integrated with respect to time to determine X-, Y-,and Z-displacements of the controller. The X-, Y-, and Z-positions maybe determined by adding the displacements to known X-, Y-, and Z-,positions at the initial instant.

STEADY STATE Y/N—This particular information indicates whether thecontroller is in a steady state, which may be defined as any position,which may be subject to change too. In a preferred embodiment the steadystate position may be one wherein the controller is held in a more orless level orientation at a height roughly even with a user's waist.

TIME SINCE LAST STEADY STATE generally refers to data related to howlong a period of time has passed since a steady state (as referencedabove) was last detected. That determination of time may, as previouslynoted, be calculated in real-time, processor cycles, or samplingperiods. The Time Since Last Steady State data time may be importantwith regard to resetting tracking of a controller with regard to aninitial point to ensure accuracy of character or object mapping in agame environment. This data may also be important with regard todetermining available actions/gestures that might be subsequentlyexecuted in a game environment (both exclusively and inclusively).

LAST GESTURE RECOGNIZED generally refers to the last gesture recognizedeither by the gesture recognizer 505 (which may be implemented inhardware or software. The identification of a last gesture recognizedmay be important with respect to the fact that a previous gesture may berelated to the possible gestures that may be subsequently recognized orsome other action that takes place in the game environment.

Time Last Gesture Recognized

The above outputs can be sampled at any time by a game program orsoftware.

In an embodiment of the present invention the mixer 408 may assign adistribution value to the tracking information 403, 405, 407 from theanalyzers 402, 404, 406. As noted above, certain sets of input controldata may be averaged. In the present embodiment, however, the inputcontrol data is assigned a value prior to its being averaged whereby theinput control data from some analyzers is of more analytical importancethan from others.

For example, the mixer 408 may require tracking information related toacceleration and steady state. The mixer 408 would then receive thetracking information 403, 405, 407 as described above. The trackinginformation may include parameters relating to acceleration and steadystate, e.g., as described above. Prior to averaging the datarepresenting this information, the mixer 408 may assign distributionvalues to tracking information data set 403, 405, 407. For example, thex- and y-acceleration parameters from the inertial analyzer 402 may beweighted at a value of 90%. The x- and y-acceleration data from theimage analyzer 406, however, may be weighted at only 10%. The acousticanalyzer tracking information 407 as it pertains to accelerationparameters may be weighted at zero percent, that is, the data has novalue.

Similarly, the Z-axis tracking information parameters from the inertialanalyzer 402 may be weighted at 10% whereas the image analyzer Z-axistracking information may be weighted at 90%. The acoustic analyzertracking information 407 may, again, be weighted at 0% value but steadystate tracking information from the acoustic analyzer 406 may beweighted at 100% with the remaining analyzer tracking information beweighted at 0%.

After the appropriate distribution weight has been assigned, the inputcontrol data may be averaged in conjunction with that weight to arriveat a weighted average input control data set that is subsequentlyanalyzed by the gesture recognizer 505 and associated with a particularaction in the game environment. The values associated may be pre-definedby the mixer 408 or by a particular game title. The values may also bethe result of the mixer 408 identifying a particular quality of datacoming from the various analyzers and thus making a dynamic adjustmentas is further discussed below. The adjustment may also be the result ofbuilding a historical knowledge base of when particular data is ofparticular value in a particular environment and/or in response to theparticularities of a given game title.

The mixer 408 may be configured to operate dynamically during game play.For example, as the mixer 408 receives various input control data, itmay recognize that certain data is consistently outside an acceptablerange or quality of data or reflects corrupt data that may be indicativeof a processing error at the related input device.

Additionally, certain conditions in a real-world environment mightchange. For example, natural light in the user's at-home gameenvironment might be increasing as the morning turns to afternooncausing problems with image data capture. Further, a neighborhood orhousehold might become noisier as the day goes on causing problems withaudio data capture. Likewise, if a user has been playing for severalhours, their reflexes may become less sharp thus causing problems withinterpretation of inertial data.

In these instances, or in any other instance wherein the quality of aparticular form of input control data is in question, the mixer 408 maydynamically reassign distribution weight to a particular set of datacoming from a particular device such that more or less importance isgiven to particular input control data as described above. Similarly,the game environment may change over the course of the game wherein theneeds of a particular game change thus requiring a reassignment of valueor need for particular input control data.

Similarly, the mixer 408 may recognize that certain data being passed onto the gesture recognizer 505 is being processed incorrectly, slowly, ornot at all based on processing errors or feedback data that may begenerated by the gesture recognizer 505. In response to this feedback orin recognition of these processing difficulties (e.g., while the imageanalysis data is within an acceptable range, errors result when anassociation is made by the gesture recognizer 505), the mixer 408 mayadjust what input control data it seeks from what analyzer and when, ifat all. The mixer 408 may further require certain analysis andprocessing of input control data by the proper analyzer before it ispassed to the mixer 408, which may re-process the data (e.g., averagethe data) such that a further layer of assurance is made that the datapassed to the gesture recognizer 505 will be processed effectively andappropriately.

In some embodiments, the mixer 408 may recognize that certain data iscorrupt, ineffective, or outside a particular variable and may call uponparticular input control data or variable related to that data such thatit may replace incorrect data or properly analyze and calculate certaindata with respect to the necessary variables.

According to embodiments of the present invention, a video game systemand method of the type described above may be implemented as depicted inFIG. 6. A video game system 600 may include a processor 601 and a memory602 (e.g., RAM, DRAM, ROM, and the like). In addition, the video gamesystem 600 may have multiple processors 601 if parallel processing is tobe implemented. The memory 602 includes data and game program code 604,which may include portions that are configured as described above.Specifically, the memory 602 may include inertial signal data 606 whichmay include stored controller path information as described above. Thememory 602 may also contain stored gesture data 608, e.g., datarepresenting one or more gestures relevant to the game program 604.Coded instructions executed on the processor 602 may implement amulti-input mixer 605, which may be configured and function as describedabove.

The system 600 may also include well-known support functions 610, suchas input/output (I/O) elements 611, power supplies (P/S) 612, a clock(CLK) 613 and cache 614. The apparatus 600 may optionally include a massstorage device 615 such as a disk drive, CD-ROM drive, tape drive, orthe like to store programs and/or data. The controller may alsooptionally include a display unit 616 and user interface unit 618 tofacilitate interaction between the controller 600 and a user. Thedisplay unit 616 may be in the form of a cathode ray tube (CRT) or flatpanel screen that displays text, numerals, graphical symbols or images.The user interface 618 may include a keyboard, mouse, joystick, lightpen or other device. In addition, the user interface 618 may include amicrophone, video camera or other signal transducing device to providefor direct capture of a signal to be analyzed. The processor 601, memory602 and other components of the system 600 may exchange signals (e.g.,code instructions and data) with each other via a system bus 620 asshown in FIG. 6.

A microphone array 622 may be coupled to the system 600 through the I/Ofunctions 611. The microphone array may include between about 2 andabout 8 microphones, preferably about 4 microphones with neighboringmicrophones separated by a distance of less than about 4 centimeters,preferably between about 1 centimeter and about 2 centimeters.Preferably, the microphones in the array 622 are omni-directionalmicrophones. An optional image capture unit 623 (e.g., a digital camera)may be coupled to the apparatus 600 through the I/O functions 611. Oneor more pointing actuators 625 that are mechanically coupled to thecamera may exchange signals with the processor 601 via the I/O functions611.

As used herein, the term I/O generally refers to any program, operationor device that transfers data to or from the system 600 and to or from aperipheral device. Every data transfer may be regarded as an output fromone device and an input into another. Peripheral devices includeinput-only devices, such as keyboards and mouses, output-only devices,such as printers as well as devices such as a writable CD-ROM that canact as both an input and an output device. The term “peripheral device”includes external devices, such as a mouse, keyboard, printer, monitor,microphone, game controller, camera, external Zip drive or scanner aswell as internal devices, such as a CD-ROM drive, CD-R drive or internalmodem or other peripheral such as a flash memory reader/writer, harddrive.

In certain embodiments of the invention, the apparatus 600 may be avideo game unit, which may include a controller 630 coupled to theprocessor via the I/O functions 611 either through wires (e.g., a USBcable) or wirelessly. The controller 630 may have analog joystickcontrols 631 and conventional buttons 633 that provide control signalscommonly used during playing of video games. Such video games may beimplemented as processor readable data and/or instructions from theprogram 604 which may be stored in the memory 602 or other processorreadable medium such as one associated with the mass storage device 615.In some embodiments, the mixer 605 may receive inputs from the analogjoystick controls 631 and the buttons 633.

The joystick controls 631 may generally be configured so that moving acontrol stick left or right signals movement along the X axis, andmoving it forward (up) or back (down) signals movement along the Y axis.In joysticks that are configured for three-dimensional movement,twisting the stick left (counter-clockwise) or right (clockwise) maysignal movement along the Z axis. These three axis—X Y and Z—are oftenreferred to as roll, pitch, and yaw, respectively, particularly inrelation to an aircraft.

The game controller 630 may include a communications interface operableto conduct digital communications with at least one of the processor602, a game controller 630 or both. The communications interface mayinclude a universal asynchronous receiver transmitter (“UART”). The UARTmay be operable to receive a control signal for controlling an operationof a tracking device, or for transmitting a signal from the trackingdevice for communication with another device. Alternatively, thecommunications interface includes a universal serial bus (“USB”)controller. The USB controller may be operable to receive a controlsignal for controlling an operation of the tracking device, or fortransmitting a signal from the tracking device for communication withanother device.

In addition, the controller 630 may include one or more inertial sensors632, which may provide position and/or orientation information to theprocessor 601 via an inertial signal. Orientation information mayinclude angular information such as a tilt, roll or yaw of thecontroller 630. By way of example, the inertial sensors 632 may includeany number and/or combination of accelerometers, gyroscopes or tiltsensors. In a preferred embodiment, the inertial sensors 632 includetilt sensors adapted to sense orientation of the game controller 632with respect to tilt and roll axes, a first accelerometer adapted tosense acceleration along a yaw axis and a second accelerometer adaptedto sense angular acceleration with respect to the yaw axis. Anaccelerometer may be implemented, e.g., as a MEMS device including amass mounted by one or more springs with sensors for sensingdisplacement of the mass relative to one or more directions. Signalsfrom the sensors that are dependent on the displacement of the mass maybe used to determine an acceleration of the game controller 630. Suchtechniques may be implemented by instructions from the game program 604which may be stored in the memory 602 and executed by the processor 601.

By way of example an accelerometer suitable as the inertial sensor 632may be a simple mass elastically coupled at three or four points to aframe, e.g., by springs. Pitch and roll axes lie in a plane thatintersects the frame, which is mounted to the game controller 630. Asthe frame (and the game controller 630) rotates about pitch and rollaxes the mass will displace under the influence of gravity and thesprings will elongate or compress in a way that depends on the angle ofpitch and/or roll. The displacement and of the mass can be sensed andconverted to a signal that is dependent on the amount of pitch and/orroll. Angular acceleration about the yaw axis or linear accelerationalong the yaw axis may also produce characteristic patterns ofcompression and/or elongation of the springs or motion of the mass thatcan be sensed and converted to signals that are dependent on the amountof angular or linear acceleration. Such an accelerometer device canmeasure tilt, roll angular acceleration about the yaw axis and linearacceleration along the yaw axis by tracking movement of the mass orcompression and expansion forces of the springs. There are a number ofdifferent ways to track the position of the mass and/or or the forcesexerted on it, including resistive strain gauge material, photonicsensors, magnetic sensors, hall-effect devices, piezoelectric devices,capacitive sensors, and the like.

In addition, the game controller 630 may include one or more lightsources 634, such as light emitting diodes (LEDs). The light sources 634may be used to distinguish one controller from the other. For exampleone or more LEDs can accomplish this by flashing or holding an LEDpattern code. By way of example, 5 LEDs can be provided on the gamecontroller 630 in a linear or two-dimensional pattern. Although a lineararray of LEDs is preferred, the LEDs may alternatively, be arranged in arectangular pattern or an arcuate pattern to facilitate determination ofan image plane of the LED array when analyzing an image of the LEDpattern obtained by the image capture unit 623. Furthermore, the LEDpattern codes may also be used to determine the positioning of the gamecontroller 630 during game play. For instance, the LEDs can assist inidentifying tilt, yaw and roll of the controllers. This detectionpattern can assist in providing a better user/feel in games, such asaircraft flying games, etc. The image capture unit 623 may captureimages containing the game controller 630 and light sources 634.Analysis of such images can determine the location and/or orientation ofthe game controller. Such analysis may be implemented by program codeinstructions 604 stored in the memory 602 and executed by the processor601. To facilitate capture of images of the light sources 634 by theimage capture unit 623, the light sources 634 may be placed on two ormore different sides of the game controller 630, e.g., on the front andon the back (as shown in phantom). Such placement allows the imagecapture unit 623 to obtain images of the light sources 634 for differentorientations of the game controller 630 depending on how the gamecontroller 630 is held by a user.

In addition the light sources 634 may provide telemetry signals to theprocessor 601, e.g., in pulse code, amplitude modulation or frequencymodulation format. Such telemetry signals may indicate which joystickbuttons are being pressed and/or how hard such buttons are beingpressed. Telemetry signals may be encoded into the optical signal, e.g.,by pulse coding, pulse width modulation, frequency modulation or lightintensity (amplitude) modulation. The processor 601 may decode thetelemetry signal from the optical signal and execute a game command inresponse to the decoded telemetry signal. Telemetry signals may bedecoded from analysis of images of the game controller 630 obtained bythe image capture unit 623. Alternatively, the apparatus 601 may includea separate optical sensor dedicated to receiving telemetry signals fromthe lights sources 634. The use of LEDs in conjunction with determiningan intensity amount in interfacing with a computer program is described,e.g., in U.S. patent application Ser. No. ______, to Richard L. Marks etal., entitled “USE OF COMPUTER IMAGE AND AUDIO PROCESSING IN DETERMININGAN INTENSITY AMOUNT WHEN INTERFACING WITH A COMPUTER PROGRAM” (AttorneyDocket No. SONYP052), filed May 4, 2006, which is incorporated herein byreference in its entirety. In addition, analysis of images containingthe light sources 634 may be used for both telemetry and determining theposition and/or orientation of the game controller 630. Such techniquesmay be implemented by instructions of the program 604 which may bestored in the memory 602 and executed by the processor 601.

The processor 601 may use the inertial signals from the inertial sensor632 in conjunction with optical signals from light sources 634 detectedby the image capture unit 623 and/or sound source location andcharacterization information from acoustic signals detected by themicrophone array 622 to deduce information on the location and/ororientation of the controller 630 and/or its user. For example,“acoustic radar” sound source location and characterization may be usedin conjunction with the microphone array 622 to track a moving voicewhile motion of the game controller is independently tracked (throughthe inertial sensor 632 and or light sources 634). In acoustic radarselect a pre-calibrated listening zone is selected at runtime and soundsoriginating from sources outside the pre-calibrated listening zone arefiltered out. The pre-calibrated listening zones may include a listeningzone that corresponds to a volume of focus or field of view of the imagecapture unit 623. Examples of acoustic radar are described in detail inU.S. patent application Ser. No. 11/381,724, to Xiadong Mao entitled“METHODS AND APPARATUS FOR TARGETED SOUND DETECTION ANDCHARACTERIZATION”, filed May 4, 2006, which is incorporated herein byreference. Any number of different combinations of different modes ofproviding control signals to the processor 601 may be used inconjunction with embodiments of the present invention. Such techniquesmay be implemented by program code instructions 604 which may be storedin the memory 602 and executed by the processor 601 and may optionallyinclude one or more instructions that direct the one or more processorsto select a pre-calibrated listening zone at runtime and filter outsounds originating from sources outside the pre-calibrated listeningzone. The pre-calibrated listening zones may include a listening zonethat corresponds to a volume of focus or field of view of the imagecapture unit 623.

The program 604 may optionally include one or more instructions thatdirect the one or more processors to produce a discrete time domaininput signal x_(m)(t) from microphones M₀ . . . M_(M), of the microphonearray 622, determine a listening sector, and use the listening sector ina semi-blind source separation to select the finite impulse responsefilter coefficients to separate out different sound sources from inputsignal x_(m)(t). The program 604 may also include instructions to applyone or more fractional delays to selected input signals x_(m)(t) otherthan an input signal x₀(t) from a reference microphone M₀. Eachfractional delay may be selected to optimize a signal to noise ratio ofa discrete time domain output signal y(t) from the microphone array. Thefractional delays may be selected to such that a signal from thereference microphone M₀ is first in time relative to signals from theother microphone(s) of the array. The program 604 may also includeinstructions to introduce a fractional time delay Δ into an outputsignal y(t) of the microphone array so that:y(t+Δ)=x(t+Δ)*b₀+x(t−1+Δ)*b₁+x(t−2+Δ)*b₂+ . . . +x(t−N+Δ)b_(N), where Ais between zero and ±1. Examples of such techniques are described indetail in U.S. patent application Ser. No. 11/381,729, to Xiadong Mao,entitled “ULTRA SMALL MICROPHONE ARRAY” filed May 4, 2006, the entiredisclosures of which are incorporated by reference.

The program 604 may include one or more instructions which, whenexecuted, cause the system 600 to select a pre-calibrated listeningsector that contains a source of sound. Such instructions may cause theapparatus to determine whether a source of sound lies within an initialsector or on a particular side of the initial sector. If the source ofsound does not lie within the default sector, the instructions may, whenexecuted, select a different sector on the particular side of thedefault sector. The different sector may be characterized by anattenuation of the input signals that is closest to an optimum value.These instructions may, when executed, calculate an attenuation of inputsignals from the microphone array 622 and the attenuation to an optimumvalue. The instructions may, when executed, cause the apparatus 600 todetermine a value of an attenuation of the input signals for one or moresectors and select a sector for which the attenuation is closest to anoptimum value. Examples of such a technique are described, e.g., in U.S.patent application Ser. No. 11/381,725, to Xiadong Mao, entitled“METHODS AND APPARATUS FOR TARGETED SOUND DETECTION” filed May 4, 2006,the disclosures of which are incorporated herein by reference.

Signals from the inertial sensor 632 may provide part of a trackinginformation input and signals generated from the image capture unit 623from tracking the one or more light sources 634 may provide another partof the tracking information input. By way of example, and withoutlimitation, such “mixed mode” signals may be used in a football typevideo game in which a Quarterback pitches the ball to the right after ahead fake head movement to the left. Specifically, a game player holdingthe controller 630 may turn his head to the left and make a sound whilemaking a pitch movement swinging the controller out to the right like itwas the football. The microphone array 620 in conjunction with “acousticradar” program code can track the user's voice. The image capture unit623 can track the motion of the user's head or track other commands thatdo not require sound or use of the controller. The sensor 632 may trackthe motion of the game controller (representing the football). The imagecapture unit 623 may also track the light sources 634 on the controller630. The user may release of the “ball” upon reaching a certain amountand/or direction of acceleration of the game controller 630 or upon akey command triggered by pressing a button on the controller 630.

In certain embodiments of the present invention, an inertial signal,e.g., from an accelerometer or gyroscope may be used to determine alocation of the controller 630. Specifically, an acceleration signalfrom an accelerometer may be integrated once with respect to time todetermine a change in velocity and the velocity may be integrated withrespect to time to determine a change in position. If values of theinitial position and velocity at some time are known then the absoluteposition may be determined using these values and the changes invelocity and position. Although position determination using an inertialsensor may be made more quickly than using the image capture unit 623and light sources 634 the inertial sensor 632 may be subject to a typeof error known as “drift” in which errors that accumulate over time canlead to a discrepancy D between the position of the joystick 630calculated from the inertial signal (shown in phantom) and the actualposition of the game controller 630. Embodiments of the presentinvention allow a number of ways to deal with such errors.

For example, the drift may be cancelled out manually by re-setting theinitial position of the controller 630 to be equal to the currentcalculated position. A user may use one or more of the buttons on thecontroller 630 to trigger a command to re-set the initial position.Alternatively, image-based drift may be implemented by re-setting thecurrent position to a position determined from an image obtained fromthe image capture unit 623 as a reference. Such image-based driftcompensation may be implemented manually, e.g., when the user triggersone or more of the buttons on the game controller 630. Alternatively,image-based drift compensation may be implemented automatically, e.g.,at regular intervals of time or in response to game play. Suchtechniques may be implemented by program code instructions 604 which maybe stored in the memory 602 and executed by the processor 601.

In certain embodiments it may be desirable to compensate for spuriousdata in the inertial sensor signal. For example the signal from theinertial sensor 632 may be oversampled and a sliding average may becomputed from the oversampled signal to remove spurious data from theinertial sensor signal. In some situations it may be desirable tooversample the signal and reject a high and/or low value from somesubset of data points and compute the sliding average from the remainingdata points. Furthermore, other data sampling and manipulationtechniques may be used to adjust the signal from the inertial sensor toremove or reduce the significance of spurious data. The choice oftechnique may depend on the nature of the signal, computations to beperformed with the signal, the nature of game play or some combinationof two or more of these. Such techniques may be implemented byinstructions of the program 604 which may be stored in the memory 602and executed by the processor 601.

The processor 601 may perform analysis of inertial signal data 606 asdescribed above in response to the data 606 and program codeinstructions of a program 604 stored and retrieved by the memory 602 andexecuted by the processor module 601. Code portions of the program 604may conform to any one of a number of different programming languagessuch as Assembly, C++, JAVA or a number of other languages. Theprocessor module 601 forms a general-purpose computer that becomes aspecific purpose computer when executing programs such as the programcode 604. Although the program code 604 is described herein as beingimplemented in software and executed upon a general purpose computer,those skilled in the art will realize that the method of task managementcould alternatively be implemented using hardware such as an applicationspecific integrated circuit (ASIC) or other hardware circuitry. As such,it should be understood that embodiments of the invention can beimplemented, in whole or in part, in software, hardware or somecombination of both.

In one embodiment, among others, the program code 604 may include a setof processor readable instructions that implement a method havingfeatures in common with the method 510 of FIG. 5B and the method 520 ofFIG. 5C or some combination of two or more of these. The program code604 may generally include one or more instructions that direct the oneor more processors to analyze signals from the inertial sensor 632 togenerate position and/or orientation information and utilize theinformation during play of a video game.

The program code 604 may optionally include processor executableinstructions including one or more instructions which, when executedcause the image capture unit 623 to monitor a field of view in front ofthe image capture unit 623, identify one or more of the light sources634 within the field of view, detect a change in light emitted from thelight source(s) 634; and in response to detecting the change, triggeringan input command to the processor 601. The use of LEDs in conjunctionwith an image capture device to trigger actions in a game controller isdescribed e.g., in U.S. patent application Ser. No. 10/759,782 toRichard L. Marks, filed Jan. 16, 2004 and entitled: METHOD AND APPARATUSFOR LIGHT INPUT DEVICE, which is incorporated herein by reference in itsentirety.

The program code 604 may optionally include processor executableinstructions including one or more instructions which, when executed,use signals from the inertial sensor and signals generated from theimage capture unit from tracking the one or more light sources as inputsto a game system, e.g., as described above. The program code 604 mayoptionally include processor executable instructions including one ormore instructions which, when executed compensate for drift in theinertial sensor 632.

In addition, the program code 604 may optionally include processorexecutable instructions including one or more instructions which, whenexecuted adjust the gearing and mapping of controller manipulations togame a environment. Such a feature allows a user to change the “gearing”of manipulations of the game controller 630 to game state. For example,a 45 degree rotation of the game controller 630 may be geared to a 45degree rotation of a game object. However this 1:1 gearing ratio may bemodified so that an X degree rotation (or tilt or yaw or “manipulation”)of the controller translates to a Y rotation (or tilt or yaw or“manipulation”) of the game object. Gearing may be 1:1 ratio, 1:2 ratio,1:X ratio or X:Y ratio, where X and Y can take on arbitrary values.Additionally, mapping of input channel to game control may also bemodified over time or instantly. Modifications may comprise changinggesture trajectory models, modifying the location, scale, threshold ofgestures, etc. Such mapping may be programmed, random, tiered,staggered, etc., to provide a user with a dynamic range ofmanipulatives. Modification of the mapping, gearing or ratios can beadjusted by the game program 604 according to game play, game state,through a user modifier button (key pad, etc.) located on the gamecontroller 630, or broadly in response to the input channel. The inputchannel may include, but may not be limited to elements of user audio,audio generated by controller, tracking audio generated by thecontroller, controller button state, video camera output, controllertelemetry data, including accelerometer data, tilt, yaw, roll, position,acceleration and any other data from sensors capable of tracking a useror the user manipulation of an object.

In certain embodiments the game program 604 may change the mapping orgearing over time from one scheme or ratio to another scheme,respectively, in a predetermined time-dependent manner. Gearing andmapping changes can be applied to a game environment in various ways. Inone example, a video game character may be controlled under one gearingscheme when the character is healthy and as the character's healthdeteriorates the system may gear the controller commands so the user isforced to exacerbate the movements of the controller to gesture commandsto the character. A video game character who becomes disoriented mayforce a change of mapping of the input channel as users, for example,may be required to adjust input to regain control of the character undera new mapping. Mapping schemes that modify the translation of the inputchannel to game commands may also change during gameplay. Thistranslation may occur in various ways in response to game state or inresponse to modifier commands issued under one or more elements of theinput channel. Gearing and mapping may also be configured to influencethe configuration and/or processing of one or more elements of the inputchannel.

In addition, a sound emitter 636, e.g., a speaker, a buzzer, a horn or apipe, may be mounted to the joystick controller 630. In certainembodiments the sound emitter may be detachably mounted to a “body” ofthe joystick controller 630. In “acoustic radar” embodiments wherein theprogram code 604 locates and characterizes sounds detected with themicrophone array 622, the sound emitter 636 may provide an audio signalthat can be detected by the microphone array 622 and used by the programcode 604 to track the position of the game controller 630. The soundemitter 636 may also be used to provide an additional “input channel”from the game controller 630 to the processor 601. Audio signals fromthe sound emitter 636 may be periodically pulsed to provide a beacon forthe acoustic radar to track location. The audio signals (pulsed orotherwise) may be audible or ultrasonic. The acoustic radar may trackthe user manipulation of the game controller 630 and where suchmanipulation tracking may include information about the position andorientation (e.g., pitch, roll or yaw angle) of the game controller 630.The pulses may be triggered at an appropriate duty cycle as one skilledin the art is capable of applying. Pulses may be initiated based on acontrol signal arbitrated from the system. The system 600 (through theprogram code 604) may coordinate the dispatch of control signals amongsttwo or more joystick controllers 630 coupled to the processor 601 toassure that multiple controllers can be tracked.

In certain embodiments, the mixer 605 may be configured to obtain inputfor controlling execution of the game program 604 using inputs receivedfrom conventional controls on the game controller 630, e.g., analogjoystick controls 631 and buttons 633. Specifically receiving the mixer605 may receive controller input information from the controller 630.The controller input information may include at least one of a)information for identifying a current position of a user-movable controlstick of the game controller in relation to a rest position of thecontrol stick, or b) information identifying whether a switch includedin the game controller is active. The mixer 605 may further receivesupplementary input information from an environment in which thecontroller 630 is being used. By way of example and without limitation,the supplementary input information may include one or more of i)information obtained from an image capture device in the environment(e.g., image capture unit 623); and/or ii) information from an inertialsensor associated with at least one of the game controller or a use(e.g., inertial sensor 632); and/or iii) acoustic information obtainedfrom an acoustic transducer in the environment (e.g., from themicrophone array 622, possibly in conjunction with an acoustic signalgenerated by the sound emitter 636).

The controller input information may also include informationidentifying whether a pressure-sensitive button is active. The mixer 605may obtain a combined input for controlling the execution of the gameprogram 604 by processing the controller input information and thesupplementary input information to yield the combined input.

The combined input may include individual merged inputs for controllingrespective individual functions during execution of the game program604. At least some of the individual merged inputs may be obtained bymerging the controller input information relative to a particularindividual function and the supplementary input information relative tothe particular individual function. The combined input may includemerged input for controlling a function during execution of the gameprogram 604, and at least some of the merged input may be obtained bymerging the controller input information relative to the function andthe supplementary input information relative to the function. In suchcases the merging may be performed by averaging a value representativeof the controller input information with a value representative of thesupplementary input information. By way of example, the value of thecontroller input information may be averaged in a one-to-one ratio withthe value of the supplementary input information. Alternatively, thecontroller input information and the supplementary input information mayeach be each assigned different weights and averaging may be performedas a weighted average of the values of controller input information andsupplementary input information in accordance with the assigned weights.

In some embodiments, a value of a first one of the controller inputinformation or the supplementary input information may be utilized asmodifying input to the game program for modifying control over a stillactive function activated in accordance with at least one of a secondone of the controller input information or the supplementary inputinformation. The supplementary input information may include inertialsensor information obtained by operation of the inertial sensor 632and/or orientation information representative of an orientation of auser-movable object. Alternatively, the supplementary input informationincludes information indicative of at least one of a position or anorientation of a user-movable object. As used here, the user-movableobject may refer to the controller 632 or an article mounted to a bodyof the controller 632 and the supplementary input information includesinformation indicative of an orientation of the user-movable object. Byway of example such orientation information may include informationindicative of at least one of pitch, yaw or roll.

In some embodiments, the combined input may be obtained by merging avalue of controller input information representative of a position of acontrol stick (e.g., one of the analog joysticks 631, with a value ofthe supplementary input information representative of the orientation ofthe user-movable object. As state above, the user-movable object mayinclude an object mounted to the game controller 630 and/or the gamecontroller 630, and the combined input may reflect an enhanced pitch upinput when the control stick is moved backward while the pitch isincreased to a positive (nose-up) value. Similarly, the combined inputmay reflect an enhanced pitch down input when the control stick is movedforward while the pitch is decreased to a negative (nose-down) value.

The combined input may be obtained by assigning the value of thecontroller input information representative of a position of the controlstick as coarse control information and assigning the value of thesupplementary input information representative of the orientation of theuser-movable object as fine control information. Alternatively, thecombined input may be obtained by assigning the value of the controllerinput information identifying whether a switch of the game controller isactive as coarse control information and assigning the value of thesupplementary input information representative of the orientation of theuser-movable object as fine control information. In addition, thecombined input may be obtained by assigning the value of thesupplementary input information representative of the orientation of theuser-movable object as coarse control information and assigning thevalue of the controller input information representative of a positionof the control stick as fine control information. Furthermore, thecombined input may also be obtained by assigning the value of thecontroller input information identifying whether a switch of the gamecontroller is active as fine control information and assigning the valueof the supplementary input information representative of the orientationof the user-movable object as coarse control information. In any or allof these cases, the combined input may be representative of the value ofthe coarse control information as adjusted a relatively small amount inaccordance with the fine control information.

In some embodiments, the combined input may be obtained by additivelycombining a value represented by the controller input information with avalue represented by the supplementary input information such that thecombined input presents a signal having a higher or lower value to thegame program 604 than either of the values of the controller inputinformation or the supplementary input information taken alone. Thecombined input may alternatively present a signal having a smoothedvalue to the game program 604, the smoothed value signal being subjectto change more slowly with time than either of the values of thecontroller input information or the supplementary input informationtaken alone. The combined input may also present a high definitionsignal having increased signal content to the game program. The highdefinition signal may be subject to change more rapidly with time thaneither of the values of the controller input information or thesupplementary input information taken alone.

Although embodiments of the present invention are described in terms ofexamples related to a video game controller 630 games, embodiments ofthe invention, including the system 600 may be used on any usermanipulated body, molded object, knob, structure, etc, with inertialsensing capability and inertial sensor signal transmission capability,wireless or otherwise.

By way of example, embodiments of the present invention may beimplemented on parallel processing systems. Such parallel processingsystems typically include two or more processor elements that areconfigured to execute parts of a program in parallel using separateprocessors. By way of example, and without limitation, FIG. 7illustrates a type of cell processor 700 according to an embodiment ofthe present invention. The cell processor 700 may be used as theprocessor 601 of FIG. 6 or the processor 502 of FIG. 5A. In the exampledepicted in FIG. 7, the cell processor 700 includes a main memory 702,power processor element (PPE) 704, and a number of synergistic processorelements (SPEs) 706. In the example depicted in FIG. 7, the cellprocessor 700 includes a single PPE 704 and eight SPE 706. In such aconfiguration, seven of the SPE 706 may be used for parallel processingand one may be reserved as a back-up in case one of the other sevenfails. A cell processor may alternatively include multiple groups ofPPEs (PPE groups) and multiple groups of SPEs (SPE groups). In such acase, hardware resources can be shared between units within a group.However, the SPEs and PPEs must appear to software as independentelements. As such, embodiments of the present invention are not limitedto use with the configuration shown in FIG. 7.

The main memory 702 typically includes both general-purpose andnonvolatile storage, as well as special-purpose hardware registers orarrays used for functions such as system configuration, data-transfersynchronization, memory-mapped J/O, and J/O subsystems. In embodimentsof the present invention, a video game program 703 may be resident inmain memory 702. The video program 703 may include inertial, image andacoustic analyzers and a mixer configured as described with respect toFIGS. 4, 5A, 5B or 5C above or some combination of these. The program703 may run on the PPE. The program 703 may be divided up into multiplesignal processing tasks that can be executed on the SPEs and/or PPE.

By way of example, the PPE 704 may be a 64-bit PowerPC Processor Unit(PPU) with associated caches L1 and L2. The PPE 704 is a general-purposeprocessing unit, which can access system management resources (such asthe memory-protection tables, for example). Hardware resources may bemapped explicitly to a real address space as seen by the PPE. Therefore,the PPE can address any of these resources directly by using anappropriate effective address value. A primary function of the PPE 704is the management and allocation of tasks for the SPEs 706 in the cellprocessor 700.

Although only a single PPE is shown in FIG. 7, some cell processorimplementations, such as cell broadband engine architecture (CBEA), thecell processor 700 may have multiple PPEs organized into PPE groups, ofwhich there may be more than one. These PPE groups may share access tothe main memory 702. Furthermore the cell processor 700 may include twoor more groups SPEs. The SPE groups may also share access to the mainmemory 702. Such configurations are within the scope of the presentinvention.

Each SPE 706 is includes a synergistic processor unit (SPU) and its ownlocal storage area LS. The local storage LS may include one or moreseparate areas of memory storage, each one associated with a specificSPU. Each SPU may be configured to only execute instructions (includingdata load and data store operations) from within its own associatedlocal storage domain. In such a configuration, data transfers betweenthe local storage LS and elsewhere in the system 700 may be performed byissuing direct memory access (DMA) commands from the memory flowcontroller (MFC) to transfer data to or from the local storage domain(of the individual SPE). The SPUs are less complex computational unitsthan the PPE 704 in that they do not perform any system managementfunctions. The SPU generally have a single instruction, multiple data(SIMD) capability and typically process data and initiate any requireddata transfers (subject to access properties set up by the PPE) in orderto perform their allocated tasks. The purpose of the SPU is to enableapplications that require a higher computational unit density and caneffectively use the provided instruction set. A significant number ofSPEs in a system managed by the PPE 704 allow for cost-effectiveprocessing over a wide range of applications.

Each SPE 706 may include a dedicated memory flow controller (MFC) thatincludes an associated memory management unit that can hold and processmemory-protection and access-permission information. The MFC providesthe primary method for data transfer, protection, and synchronizationbetween main storage of the cell processor and the local storage of anSPE. An MFC command describes the transfer to be performed. Commands fortransferring data are sometimes referred to as MFC direct memory access(DMA) commands (or MFC DMA commands).

Each MFC may support multiple DMA transfers at the same time and canmaintain and process multiple MFC commands. Each MFC DMA data transfercommand request may involve both a local storage address (LSA) and aneffective address (EA). The local storage address may directly addressonly the local storage area of its associated SPE. The effective addressmay have a more general application, e.g., it may be able to referencemain storage, including all the SPE local storage areas, if they arealiased into the real address space.

To facilitate communication between the SPEs 706 and/or between the SPEs706 and the PPE 704, the SPEs 706 and PPE 704 may include signalnotification registers that are tied to signaling events. The PPE 704and SPEs 706 may be coupled by a star topology in which the PPE 704 actsas a router to transmit messages to the SPEs 706. Alternatively, eachSPE 706 and the PPE 704 may have a one-way signal notification registerreferred to as a mailbox. The mailbox can be used by an SPE 706 to hostoperating system (OS) synchronization.

The cell processor 700 may include an input/output (J/O) function 708through which the cell processor 700 may interface with peripheraldevices, such as a microphone array 712 and optional image capture unit713 and a game controller 730. The game controller unit may include aninertial sensor 732, and light sources 734. In addition an ElementInterconnect Bus 710 may connect the various components listed above.Each SPE and the PPE can access the bus 710 through a bus interfaceunits BIU. The cell processor 700 may also includes two controllerstypically found in a processor: a Memory Interface Controller MIC thatcontrols the flow of data between the bus 710 and the main memory 702,and a Bus Interface Controller BIC, which controls the flow of databetween the J/O 708 and the bus 710. Although the requirements for theMIC, BIC, BIUs and bus 710 may vary widely for differentimplementations, those of skill in the art will be familiar theirfunctions and circuits for implementing them.

The cell processor 700 may also include an internal interrupt controllerIIC. The IIC component manages the priority of the interrupts presentedto the PPE. The IIC allows interrupts from the other components the cellprocessor 700 to be handled without using a main system interruptcontroller. The IIC may be regarded as a second level controller. Themain system interrupt controller may handle interrupts originatingexternal to the cell processor.

In embodiments of the present invention, certain computations, such asthe fractional delays described above, may be performed in parallelusing the PPE 704 and/or one or more of the SPE 706. Each fractionaldelay calculation may be run as one or more separate tasks thatdifferent SPE 706 may take as they become available.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A” or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

1. A method of obtaining input for controlling execution of a gameprogram, comprising: receiving controller input information from auser-manipulable game controller, the controller input informationincluding at least one of a) information for identifying a currentposition of a user-movable control stick of the game controller inrelation to a rest position of the control stick, or b) informationidentifying whether a switch included in the game controller is active;receiving supplementary input information from an environment in whichthe controller is being used; and obtaining a combined input forcontrolling the execution of the game program by processing thecontroller input information and the supplementary input information toyield the combined input.
 2. The method as claimed in claim 1, whereinthe combined input includes individual merged inputs for controllingrespective individual functions during execution of the game program,and at least some of the individual merged inputs are obtained bymerging the controller input information relative to a particularindividual function and the supplementary input information relative tothe particular individual function.
 3. The method as claimed in claim 1,wherein the combined input includes merged input for controlling afunction during execution of the game program, and at least some of themerged input is obtained by merging the controller input informationrelative to the function and the supplementary input informationrelative to the function.
 4. The method as claimed in claims 2 or 3,wherein the merging is performed by averaging a value representative ofthe controller input information with a value representative of thesupplementary input information.
 5. The method as claimed in claim 4,wherein the value of the controller input information is averaged in aone-to-one ratio with the value of the supplementary input information.6. The method as claimed in claim 4, wherein the controller inputinformation and the supplementary input information are each assigneddifferent weights and the step of averaging is performed as a weightedaverage of the values of controller input information and supplementaryinput information in accordance with the assigned weights.
 7. The methodas claimed in claim 1, further comprising utilizing a value of a firstone of the controller input information or the supplementary inputinformation as modifying input to the game program for modifying controlover a still active function activated in accordance with at least oneof a second one of the controller input information or the supplementaryinput information.
 8. The method as claimed in claims 2, 3 or 7, whereinthe supplementary input information includes at least one of inertialsensor information obtained by operation of an inertial sensor ororientation information representative of an orientation of auser-movable object.
 9. The method as claimed in claim 8, wherein theinertial sensor is mounted to the game controller and the inertialsensor includes at least one of an accelerometer or a gyroscope.
 10. Themethod as claimed in claim 2 or 8, wherein the supplementary inputinformation includes information indicative of at least one of aposition or an orientation of a user-movable object.
 11. The method asclaimed in claim 10, wherein the user-movable object includes at leastone of the game controller or an article mounted to a body of the gamecontroller and the supplementary input information includes informationindicative of an orientation of the user-movable object.
 12. The methodas claimed in claim 10, wherein the supplementary input informationincludes information indicative of at least one of pitch, yaw or roll.13. The method as claimed in claim 12, wherein the supplementary inputinformation includes information indicative of pitch, yaw and roll. 14.The method as claimed in any one of claims 10 through 13, wherein thecombined input is obtained by merging a value of controller inputinformation representative of a position of the control stick with avalue of the supplementary input information representative of theorientation of the user-movable object.
 15. The method as claimed inclaim 14, wherein the user-movable object includes at least one of anobject mounted to the game controller or the game controller, and thecombined input reflects an enhanced pitch up input when the controlstick is moved backward while the pitch is increased to a positive(nose-up) value.
 16. The method as claimed in claim 15, wherein theuser-movable object includes at least one of an object mounted to thegame controller or the game controller, and the combined input reflectsan enhanced pitch down input when the control stick is moved forwardwhile the pitch is decreased to a negative (nose-down) value.
 17. Themethod as claimed in claim 14, wherein the combined input is obtained byassigning the value of the controller input information representativeof a position of the control stick as coarse control information andassigning the value of the supplementary input informationrepresentative of the orientation of the user-movable object as finecontrol information, wherein the combined input is representative of thevalue of the coarse control information as adjusted a relatively smallamount in accordance with the fine control information.
 18. The methodas claimed in claim 14 or claim 17, wherein the combined input isobtained by assigning the value of the controller input informationidentifying whether a switch of the game controller is active as coarsecontrol information and assigning the value of the supplementary inputinformation representative of the orientation of the user-movable objectas fine control information, wherein the combined input isrepresentative of the value of the coarse control information asadjusted a relatively small amount in accordance with the fine controlinformation.
 19. The method as claimed in claim 14 or claim 18, whereinthe combined input is obtained by assigning the value of thesupplementary input information representative of the orientation of theuser-movable object as coarse control information and assigning thevalue of the controller input information representative of a positionof the control stick as fine control information, wherein the combinedinput is representative of the value of the coarse control informationas adjusted a relatively small amount in accordance with the finecontrol information.
 20. The method as claimed in claim 14, 17 or 19,wherein the combined input is obtained by assigning the value of thecontroller input information identifying whether a switch of the gamecontroller is active as fine control information and assigning the valueof the supplementary input information representative of the orientationof the user-movable object as coarse control information, wherein thecombined input is representative of the value of the coarse controlinformation as adjusted a relatively small amount in accordance with thefine control information.
 21. The method as claimed in claims 1, 2 or 3,wherein the combined input is obtained by additively combining a valuerepresented by the controller input information with a value representedby the supplementary input information such that the combined inputpresents a signal having a higher value to the game program than eitherof the values of the controller input information or the supplementaryinput information taken alone.
 22. The method as claimed in claims 1, 2or 3, wherein the combined input is obtained by subtractively combininga value represented by the controller input information with a valuerepresented by the supplementary input information such that thecombined input presents a signal having a lower value to the gameprogram than either of the values of the controller input information orthe supplementary input information taken alone.
 23. The method asclaimed in claims 1, 2 or 3, wherein the combined input presents asignal having a smoothed value to the game program, the smoothed valuesignal being subject to change more slowly with time than either of thevalues of the controller input information or the supplementary inputinformation taken alone.
 24. The method as claimed in claims 1, 2 or 3,wherein the combined input presents a high definition signal havingincreased signal content to the game program, the high definition signalbeing subject to change more rapidly with time than either of the valuesof the controller input information or the supplementary inputinformation taken alone.
 25. The method as claimed in any one of claims1, 2 or 3, wherein the supplementary input information includes acousticinformation obtained from an acoustic transducer in the environment. 26.The method as claimed in any one of claims 1, 2 or 3, wherein thecontroller input information includes information identifying whether apressure-sensitive button is active.
 27. The method as claimed in anyone of claims 1, 2 or 3, wherein the supplementary input informationincludes at least one of: i) information obtained from an image capturedevice in the environment, ii) information from an inertial sensorassociated with at least one of the game controller or the user, or iii)information from an acoustic transducer in the environment.
 28. Themethod as claimed in any one of claims 1, 2 or 3, wherein thesupplementary input information includes information obtained from animage capture device in the environment, information from an inertialsensor associated with at least one of the game controller or the user,and information from an acoustic transducer in the environment.